1. Field of the Invention
This invention relates to forward looking infrared (FLIR) and related systems and, more specifically, to systems and methods for applying interleaved sampling when using scanned time delay and integrate (TDI) focal plane array (FPA) detectors to obtain high resolution without objectionable artifacts.
2. Brief Description of the Prior Art
High sensitivity and resolution in FLIR systems using scanned non-TDI FPA detectors, while eliminating artifacts, has been achieved in the prior art as evidenced by U.S. Pat. No. 5,140,147 for "INTRAFIELD INTERLEAVED SAMPLE VIDEO PROCESSOR/REFORMATTER" of James S. Barnett, the contents of which are incorporated herein by reference. However, current TDI detector architectures do not permit both rectangular and interleaved sampling.
Non-TDI scanned FPA detectors typically contain 1 to 4 columns of detector elements as shown in FIG. 1. Typical non-TDI and TDI FPA detectors are designed to take two samples while the array is scanned across a distance equal to the instantaneous field of view (IFOV) or width of a single element. FIG. 1 shows an example of a four column non-TDI detector with an intercolumn spacing of N IFOVs. N is frequently an integer but this is not a requirement. More generally, M=NW where M is an integer number of sample spaces between columns and N is not necessarily an integer. However, the principles involved are generally applicable to other configurations of FPA detectors.
Current FPA detectors are typically designed to operate at a 30 Hz scan rate to minimize signal bandwidth and to provide a rectangular sample pattern as shown in FIG. 2. Current FPA designs were primarily influenced by the perceived needs of automatic target cuers, classifiers and recognizers. Such target identifiers demand very high resolution, non-interlaced, "snapshot" images with high sensitivity. Their operation is independent of image displays and they usually operate at scan rates of 30 Hz or less.
Many applications of FPA detectors require that the FLIR system display images and automatically identify and track targets. In scenarios involving high angular rates, automatic target trackers require 60 Hz image update rates from the FPA. Standard television monitors display interlaced video at a 60 Hz field rate and 30 Hz frame rate as shown in FIGS. 3A to 3C. Display of current 30 Hz, non-interlaced FPA video on standard 60 Hz television format displays causes objectionable flicker, reduces useful resolution and introduces severe artifacts for moving scenes or for moving objects within a still scene.
FIG. 4 shows the functional components of a typical FPA FLIR system. Various configurations and implementation details are feasible. Optics 1 collect infrared energy from the scene. To simplify detector fabrication and to reduce noise sources, FLIR systems typically sample all elements simultaneously as the scanner 2 moves the image across the FPA detector 3. FPA timing electronics 4 typically derive sampling and multiplexing control signals from scene position information provided by a scan position sensor and/or from fixed frequency clocks. The FPA converts infrared energy to a sampled electrical signal. The signal from each detector element shown in FIG. 1 is amplified and sent to a multiplexer 5. The multiplexer reduces the number of outputs required from the detector. A FPA detector with 960 elements typically is multiplexed to 16 outputs. Analog-to-digital conversion and digital multiplexing 6 provide four streams of digital information, each representing one of four columns of detectors shown in FIG. 1.
Since the columns are spatially separated, the Mth sample from detector elements in column 1 is misaligned by 2N sample intervals, for a sampling density of two samples/IFOV, compared to the Mth sample from detector elements in column 2. N is the center-to-center spacing (in IFOVs) between columns. Total misalignment between columns 1 and 4 equals 6N sample times at two samples/IFOV. Column alignment circuits 7 delay the samples from each of the four columns by the proper time (number of samples) to align them spatially for display. N equals 4 for a typical non-TDI detector. Table 1 shows the number of sample delays required to align detector columns for three values of N.
TABLE 1 __________________________________________________________________________ REQUIRED SAMPLE DELAYS TO ALIGN NON-TDI DETECTOR COLUMNS FOR RECTANGULAR SAMPLE PATTERNS AT TWO SAMPLES PER IFOV N Column 1 Delay Column 2 Delay Column 3 Delay Column 4 Delay __________________________________________________________________________ 5 30 20 10 0 4 24 16 8 0 3 18 12 6 0 __________________________________________________________________________
These circuits, shown in FIG. 5, typically use first-in, first-out (FIFO) memories. They delay column 1 (elements 1, 5, 9, etc.) by 6N samples, column 2 by 4N samples and column 3 by 2N samples with respect to samples from column 4 when the FPA shown in FIG. 1 is scanned from left to right. This produces the rectangular sampling pattern shown in FIG. 2. If the image is scanned from right to left, these circuits delay column 4 by 6N, column 3 by 4N and column 2 by 2N samples with respect to column 1. The same function also could be embedded into the addressing scheme for the reformatter function 9 shown in FIG. 4.
Image processor circuits 8 perform various functions to enhance image quality for display and for automatic target trackers and identifiers. Reformatter circuits 9 store the sampled FPA data in memory in FPA detector output format (4 vertical columns), then read it out in the single line interlaced television format shown in FIGS. 3A to 3C. FIG. 6 is a block diagram of a reformatter that uses two-port video random access memories (VRAMs) as the FPA sampled data storage medium. Pixel selector logic 14 controls a 4:1 multiplexer function 13, which constructs a TV format video stream from the VRAM 12 outputs. If the FPA were scanned from top to bottom rather than horizontally across the scene, reformatter memory storage requirements would be reduced since FPA output data would be in horizontal format.
Video generator circuits 10 convert digital FLIR data to standard analog composite television video signals. FIG. 7 is a block diagram of a typical video generator circuit. A high speed digital-to-analog (D/A) converter 15 converts the digital output from the reformatter into an analog signal. The low pass filter circuit 16 removes signal content at frequencies greater than one-half the sample frequency. It also provides the desired frequency domain transfer function. FIG. 8 shows a typical low pass filter response. Video buffer circuits 17 add standard vertical and horizontal synchronization signals and provide proper voltage levels and impedances to the monitor.
Simply doubling the FPA scan rate would eliminate moving image flicker and artifacts from the display and would benefit automatic target trackers. However, 30 Hz FPA data rates can be as high as 960 million bits/second for 12 bit data words. Doubling scan rate would double data rates, thereby significantly increasing the cost, power and size of FLIR system electronics and making most such systems economically unfeasible under present technology. Doubling the scan rate also would reduce FPA detector sensitivity by the square root of two. If the number of samples were reduced by a factor of two, while the scan rate were doubled, total data rate and sensitivity would remain essentially constant. However, horizontal resolution would be reduced by a factor of two.
Modifying the FPA multiplexer by adding a mode where only half the channels are output during each 60 Hz field maintains displayed resolution but reduces effective sensitivity. Interlaced multiplexing reduces sensitivity because only half the available detector elements are used and because each detector element has one-half the time to integrate infrared energy that it would have at 30 Hz.
The above identified U.S. Pat. No. 5,140,147 identifies an interleaved sampling algorithm and implementation that overcomes these problems for non-TDI FPA detectors where N in FIG. 1 is an integer. This innovative approach takes advantage of the fact that non-TDI detectors nominally designed to operate at two samples/IFOV can be sampled at certain frequencies other than 2.0 samples per IFOV. The following equation shows the allowed spatial sample frequencies for such a non-TDI FPA: EQU Spatial Sample Frequency=[(2 * N)-K]* 2/(2 * N * I)
where:
SSF=Spatial Sample Frequency and is measured in samples/IFOV PA1 I=1 for Rectangular Sampling PA1 I=2 for Interleaved Sampling PA1 K=any integer for Rectangular Sampling PA1 K=odd integers for Interleaved Sampling PA1 DSF=I.times.SSF PA1 DSF=SSF for I=1 (Rectangular Sampling) PA1 DSF=2 * SSF for I=2 (Interleaved Sampling)
Displayed sample frequency (DSF) is the apparent sample frequency displayed on a standard monitor as a result of the FPA sampling and reformatting algorithms. The following equations show the relationship between SSF and DSF:
If "I" equals one, any integer value of "K" will result in the rectangular spatial sampling pattern shown in FIG. 2. There is an equal integer number of samples between each detector column. Table 2 shows several possible values of SSF for rectangular sampling in addition to the normal value of 2.00. However, rectangular sampling patterns are not formed if "K" is an odd integer and I=2.
TABLE 2 ______________________________________ PERMITTED SPATIAL SAMPLING FREQUENCIES FOR NON-TDI DETECTORS THAT FORM A RECTANGULAR SAMPLING PATTERN. Displayed Value of K Value of I Samples/IFOV Samples/IFOV ______________________________________ -1 1 2.25 2.25 0 1 2.00 2.00 1 1 1.75 1.75 2 1 1.50 1.50 3 1 1.25 1.25 4 1 1.00 1.00 5 1 0.75 0.75 ______________________________________
Table 3 lists the SSF for several values of K, for N=4. SSF is about one half the nominal value of 2.00 for K=-1 or for K=1, while DSF is approximately 2.00. These SSFs can therefore be achieved at a 60 Hz scan rate with little impact on detector sensitivity, FLIR resolution or FLIR video processor electronics size, weight or power.
TABLE 2 ______________________________________ PERMITTED SPATIAL SAMPLING FREQUENCIES FOR NON-TDI DETECTORS THAT FORM A INTERLEAVED SAMPLING PATTERN. Displayed Value of K Value of I Samples/IFOV Samples/IFOV ______________________________________ -3 2 1.375 2.75 -1 2 1.125 2.25 1 2 0.875 1.75 3 2 0.625 1.25 ______________________________________
FIG. 9 shows an overlay of the sampling pattern with respect to the non-TDI array for K=1 and I=2. There is an equal integer number of samples between every other column. Samples between adjacent columns have a half-IFOV spatial offset. If the external column alignment circuits are programmed to delay column 1 video by 10 sample times, column 2 by 7 sample times and column 3 by 3 sample times, columns 1 and 3 will be aligned with each other but offset by one-half IFOV from columns 2 and 4. FIG. 10 shows the resulting pattern.
Table 4 shows the sample delays that must be programmed into the column alignment circuit illustrated in FIG. 5 to obtain the offset sampling pattern shown in FIG. 10 for several values of K and N.
TABLE 4 ______________________________________ REQUIRED SAMPLE DELAYS TO ALIGN NON-TDI DETECTOR COLUMNS FOR INTERLEAVED SAMPLE PATTERNS. Column 1 Column 2 Column 3 Column 4 N K Delay Delay Delay Delay ______________________________________ 4 -3 16 11 5 0 4 -1 13 9 4 0 4 1 10 7 3 0 3 -1 10 7 3 0 4 3 7 5 2 0 ______________________________________
The reformatter shown in FIG. 6 is normally programmed to construct each TV line from a sequence of M samples from a single detector element. The reformatter forms a TV frame of L horizontal TV lines, each line containing M samples, by reading M samples from L/2 detectors at a 60 Hz rate as shown in FIGS. 3A to 3C. To display FLIR images in interleaved format, the reformatter reads approximately M/2 samples from L detectors each TV field. FIGS. 11, 12 and 13 illustrate the required algorithm.
Line 1 of TV field A is constructed by reading the first sample from detector 1, then the second sample from detector 2, then the third sample from detector 1, etc. The second line (TV line 3) 2 of TV field A is formed using samples from detectors 3 and 4. The first line of field B (TV line 2) is formed by detectors 2 and 3, the second line (TV line 4) from detectors 4 and 5, etc. This provides the effective vertical interlace between TV fields A and B shown in FIGS. 3A to 3C and 13.
Since every other TV format sample is displayed at a vertical offset of one-half IFOV from its true position, interleaved sampling creates an artifact. The displayed image of a horizontal edge will have serrations along the edge one sample wide and one sample deep. Vertical edges are not affected. This artifact is independent of motion and is easily removed by a one-dimensional low pass filter with a notch at one-half the sample frequency. FIG. 8 shows an example of such a filter.
FIG. 14 shows the architecture of a typical TDI detector. Four "super columns" each containing 4 TDI subelements (A, B, C and D), replace the four columns of detector elements in FIG. 1. FIG. 14 shows twenty channels out of a typical 480 or 960. FIG. 15 illustrates internal TDI function. Delay circuits 19, 20 and 21 time-delay analog domain samples from each of the four TDI subelements within a single FPA channel. Next, the summer 22 adds (integrates) samples from the four subelements into a single output value. It then sends the integrated sample to the FPA output multiplexer. TDI of four subelements improves FPA signal-to-noise ratio by a factor of up to 2:1 compared to the non-TDI detector shown in FIG. 1. The TDI function can be implemented using charge coupled devices (CCD), delay lines or charge steering approaches.
The four "OR" functions 18 in FIG. 15 allow two-way scan operation. When the array shown in FIG. 14 is scanned left-to-right, the "OR" functions steer subelements A to the 6N sample delay function. Subelement D has zero sample delay. When the array is scanned right-to-left, subelement D is delayed 6N sample periods and subelement a has zero delay. "OR" circuits also select the proper delay element for subelements B and C, depending upon the scan direction.
Current TDI detectors are typically fabricated with equal element spacings of 1.5 to 2.5 IFOVs between all subelements, including those at the boundary between super-columns. Table 5 shows the total delay required, in numbers of samples at two samples per IFOV, to align each subelement within a single detector channel for various interelement spacings.
TABLE 5 ______________________________________ TOTAL SUBELEMENT DELAY REQUIRED (TWO SAMPLES/IFOV) Spacing: N A B C D ______________________________________ 1.5 IFOV's 1.5 9 6 3 0 2.0 IFOV's 2.0 12 8 4 0 2.5 IFOV's 2.5 15 10 5 0 3.0 IFOV's 3.0 18 12 6 0 4.0 IFOV's 4.0 24 16 8 0 ______________________________________
External column alignment circuits shown in FIG. 5 align TDI FPA detector super-columns. TDI detectors typically require greater total delay than non-TDI FPAs. Detector geometry and TDI circuit implementation (FIG. 15) have limited TDI detectors to a single sample frequency, typically two samples per detector IFOV. This prohibits the use of interleaved sampling techniques to eliminate motion artifacts. Doubling the scan rate would generally result in a prohibitive cost, size and power increase for the FLIR video processor electronics shown in FIG. 4.